Fun with Asynchronous Vision Sensors and Processing

This paper provides a personal perspective on our group’s efforts in building event-based vision sensors, algorithms, and applications over the period 2002-2012. Some recent advances from other groups are also briefly described. When Mahowald and Mead built the first silicon retina with asynchronous digital output around 1992 [1], conventional CMOS active pixel sensors (APS) were still research chips. It required the investment by industry of about a billion dollars to bring CMOS APS to high volume production. So it is no surprise that while the imager community has been consumed by the megapixel race to make nice photos, cameras that mimic more closely how the eye works have taken a long time to come to a useful form. These “silicon retinas” are much more complex at the pixel level than APS cameras and they pay the price in terms of fill factor and pixel size; machine vision cameras with capability of synchronous global electronic shutter are about 5um. Silicon retina pixels are roughly 10 times the area of a machine vision camera pixel. So why are silicon retinas still interesting? Mostly because of the high cost at the system level of processing the highly redundant data from conventional cameras, and the fixed latencies imposed by the frame intervals. High performance activity driven event-based sensors could greatly benefit applications in real time robotics, where just as in nature, latency and power are very important [2,5,9,10].